Skip to content

Buffer up to 2048 doc ids in for_each_docset_buffered - #2944

Draft
fulmicoton-dd wants to merge 1 commit into
mainfrom
larger-collect-block-buffer
Draft

Buffer up to 2048 doc ids in for_each_docset_buffered#2944
fulmicoton-dd wants to merge 1 commit into
mainfrom
larger-collect-block-buffer

Conversation

@fulmicoton-dd

@fulmicoton-dd fulmicoton-dd commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator

What

In the no-score collection path (Weight::for_each_no_scorefor_each_docset_buffered),
matched doc ids were handed to the collector's collect_block one
COLLECT_BLOCK_BUFFER_LEN (64) block at a time. For aggregations this is the dominant path
(requires_scoring() == false), and 64 docs per collect_block call under-amortizes the
per-call overhead (virtual dispatch + aggregation bookkeeping).

for_each_docset_buffered now owns a 2048-element heap buffer and fills it through
successive fill_buffer calls over 64-element windows, flushing a single larger block to
collect_block. fill_buffer keeps its COLLECT_BLOCK_BUFFER_LEN (64) window contract, so
no DocSet implementation changes — only how much we accumulate before flushing.

Notes

  • The large buffer is allocated with Box::new_zeroed_slice(..).assume_init() to zero directly
    on the heap (no 2048-element stack temporary). This API is stable since 1.92.0, hence the
    MSRV bump 1.86 → 1.92.
  • Audited every collect_block implementation: none assume a max slice length of 64 (they
    delegate or iterate the slice; filter wrappers use a growable Vec). Doc comments on
    COLLECT_BLOCK_BUFFER_LEN and SegmentCollector::collect_block updated accordingly — the
    block handed to collect_block may now exceed 64.
  • Uses unsafe to reinterpret each 64-element chunks_exact_mut window as &mut [DocId; 64]
    for the fill_buffer call (sound: 2048 is a multiple of 64).

Benchmarks

cargo bench --bench agg_bench on the full (1M-doc) input, single segment, no deletes — so every bench exercises the for_each_no_scorecollect_block path. Reported value is the minimum of 3 runs (most stable estimator; the per-run median swung ±5–20% on its own). Apple Silicon (M-series), REUSE_AGG_BENCH_INDEX so both sides hit the identical index.

Summary: median -1.7%, mean -1.6% across 47 aggregations. Negative = faster. Broad improvement on the cheap full-scan aggregations (where per-collect_block overhead dominated); a handful of heavy order_by_card / top_hits benches regress and are within this harness's run-to-run noise.

Per-benchmark median (min of 3), sorted by delta
benchmark before after delta
histogram_hard_bounds 1.602 ms 1.428 ms -10.9%
cardinality_agg_low_card 1.192 ms 1.070 ms -10.2%
filter_agg_all_query_count_agg 5.093 ms 4.801 ms -5.7%
histogram 2.878 ms 2.715 ms -5.7%
terms_many_order_by_term 5.514 ms 5.265 ms -4.5%
range_agg_with_term_agg_status 6.409 ms 6.146 ms -4.1%
filter_agg_term_query_count_agg 7.519 ms 7.220 ms -4.0%
average_u64 3.025 ms 2.908 ms -3.9%
average_f64 3.180 ms 3.060 ms -3.8%
average_f64_u64 5.904 ms 5.683 ms -3.8%
range_agg 3.348 ms 3.224 ms -3.7%
range_agg_with_avg_sub_agg 7.145 ms 6.890 ms -3.6%
terms_status_with_cardinality_agg 3.358 ms 3.246 ms -3.3%
terms_all_unique 6.164 ms 5.965 ms -3.2%
avg_and_range_with_avg_sub_agg 10.060 ms 9.740 ms -3.2%
terms_all_unique_with_avg_sub_agg 19.049 ms 18.452 ms -3.1%
terms_status_with_avg_sub_agg 5.472 ms 5.312 ms -2.9%
stats_f64 3.160 ms 3.068 ms -2.9%
filter_agg_all_query_with_sub_aggs 10.079 ms 9.827 ms -2.5%
extendedstats_f64 3.324 ms 3.242 ms -2.5%
histogram_with_avg_sub_agg 9.345 ms 9.123 ms -2.4%
histogram_with_term_agg_status 9.441 ms 9.230 ms -2.2%
terms_7 2.344 ms 2.300 ms -1.9%
filter_agg_term_query_with_sub_aggs 12.473 ms 12.257 ms -1.7%
terms_150_000 6.462 ms 6.350 ms -1.7%
range_agg_with_term_agg_many 12.541 ms 12.350 ms -1.5%
percentiles_f64 6.476 ms 6.381 ms -1.5%
terms_status_with_terms_zipf_1000_sub_agg 4.112 ms 4.053 ms -1.4%
terms_status_with_histogram 4.873 ms 4.828 ms -0.9%
cardinality_agg 13.139 ms 13.047 ms -0.7%
terms_many_with_avg_sub_agg 17.935 ms 17.819 ms -0.6%
cardinality_agg_high_card 65.139 ms 64.745 ms -0.6%
terms_many_top_1000 9.417 ms 9.361 ms -0.6%
composite_term_many_page_1000 28.273 ms 28.197 ms -0.3%
composite_term_many_page_1000_with_avg_sub_agg 29.137 ms 29.147 ms +0.0%
terms_zipf_1000_with_terms_status_sub_agg 12.853 ms 12.886 ms +0.3%
terms_100_buckets_with_cardinality_agg 50.915 ms 51.058 ms +0.3%
terms_zipf_1000_with_histogram 23.664 ms 23.780 ms +0.5%
composite_histogram_calendar 27.774 ms 27.930 ms +0.6%
composite_term_few 13.571 ms 13.654 ms +0.6%
terms_zipf_1000_with_avg_sub_agg 9.544 ms 9.672 ms +1.3%
terms_many_json_mixed_type_with_avg_sub_agg 27.830 ms 28.248 ms +1.5%
composite_histogram 17.038 ms 17.488 ms +2.6%
terms_many_with_top_hits 92.988 ms 95.822 ms +3.0%
terms_zipf_1000 2.211 ms 2.317 ms +4.8%
terms_many_with_single_term_2_order_by_card 49.301 ms 51.690 ms +4.8%
terms_many_with_single_term_order_by_card 74.692 ms 81.090 ms +8.6%

Tests

cargo build, cargo clippy --all-targets (no new warnings), and
cargo test --lib {aggregation, query::, collector::} all pass.

The no-score collection path (Weight::for_each_no_score) handed the
collector's collect_block one COLLECT_BLOCK_BUFFER_LEN (64) block at a
time. For aggregations this is the dominant path, and 64 docs per
collect_block under-amortizes the per-call overhead.

for_each_docset_buffered now owns a 2048-element heap buffer and fills
it through successive fill_buffer calls over 64-element windows, flushing
a single larger block to collect_block. fill_buffer keeps its 64-element
window contract, so no DocSet implementation changes.

The buffer is allocated with Box::new_zeroed_slice (stable since 1.92,
hence the MSRV bump) to zero directly on the heap.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant